feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517) - #1289
feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517)#1289localai-bot wants to merge 20 commits into
Conversation
…e FP8 W8A8 seam (#517) The 23 Mamba2 layers were the last host bounce of a decode step, and they were the expensive one. The host arm reaches its two projections through `Linear(..., const NemotronHOwned&)`, whose `DenseFor` calls `NemotronHOwned::DenseBf16()` -- a FULL dequant of the fp8 tower into a fresh bf16 buffer ON EVERY CALL. That is 23 x (10304x2688 + 2688x4096) = 890e6 elements re-expanded per token, plus a download of the normed hidden and an upload of the mixer output per layer. The block moves whole or not at all: `mixer.in_proj` produces the fused `zxbcdt` that the causal conv and the SSD scan both consume (mamba_mixer2.py:550, split :692-696), so there is no intermediate landing in which the conv is on the device and `in_proj` is not. That is why the shared FP8 W8A8 linear seam had to be extracted first (#940, `dense_fp8_gemm.h`), and this change routes through it rather than re-typing its entry points. `NemotronHMamba2MixerDevice` mirrors the host arm statement for statement -- same vt:: ops, same order, same dtypes, different backend, which is the property A2-R established and the numeric gate reads: in_proj (FP8 W8A8) -> QkvSplit(z | xBC | dt) -> CausalConv1dFwd(silu) -> QkvSplit(x | B | C) -> Mamba2ChunkScan -> RmsNormGatedGroup(n_groups) -> out_proj (FP8 W8A8) The one substitution is the split: the host arm copies columns with `SliceCols` because `vt::Mamba2ChunkScan` validates every operand contiguous, and `vt::QkvSplit` is exactly that copy on the device. Both forwards select it at runtime, never by preprocessor: `MambaIsFp8` names only `NemotronHWeightForm` and `vt::OpRegistered` is the op table's own answer, so a dense NemotronH or a device without the fp8 pair keeps the host bounce. The paged arm hands the mixer the recurrent rows `vt::GdnStateGather` already gathered and zeroed and it advances them IN PLACE, so the two downloads, the host mixer and the two uploads per layer are gone; A2-P's gather/scatter pair and its `has_initial = true` reasoning are unchanged. The `ssm_dtype == f32` term in the paged selection is not decoration. The gather widens the page into an f32 working buffer by op contract and the host arm then narrows it back to `ssm_dtype`, so on a checkpoint whose `mamba_ssm_cache_dtype` is not f32 the two arms would round differently and the per-block numeric gate would be comparing two different computations. The device residency is built on first use and keyed on a `ResidentSlot` the weights own, never on an address (#237): across two engine builds in one process an address-keyed cache returns the previous engine's device pointers, which is plausible wrong values rather than a crash. The e4m3 staging copy is released as soon as `ResidentFp8` has uploaded it, so the conversion peaks at one projection rather than at the whole 890 MB tower. `dense_fp8::ResidentFp8` still does not account its own upload -- that is #974, and this change does not touch the shared header -- so A2-Q1 accounts what IT uploads at the site that causes it, through the same `load_stats::AddDeviceUpload` `ResidentWeight` and `ResidentNvfp4` call. That counter is also the instrument the new residency case reads: an arm that re-uploaded the tower every step returns identical numbers to one that uploads it once. G-SAFE is untouched. All three clauses of the interlock stand, this change creates no paging, no carried state and no batching that A2-P did not already create, and `num_reqs <= 1` remains A2-B's to remove. `tests/vllm/models/test_nemotron_h_mamba_device.cpp` is the cheap arm in front of the real-checkpoint gate, for the same reason A2-Q2a's device MoE file exists: `BuildTiny` is all-`kDense`, so nothing in the existing device suite could reach the FP8 W8A8 arm at all. The two arms are NOT bit-comparable by construction -- the host reference is W8A16 and the device arm is W8A8 as vLLM is -- so every band is measured in the run against a defect the fixture separates, and the element count is asserted against the geometry so a maximum over zero elements cannot read as agreement. `scripts/nemotron-h-a2q1-dgx-gate.sh` records the GB10 recipe rather than leaving it to be retyped, including the three environment facts that have each voided a run before: the CUDA lane is `sbsa`, `nvcc --version` is not a sufficient postcondition, and `/workspace` is CIFS and refuses symlinks. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
|
Evidence status, so nobody reads silence as a result. Two leases are out for the gates this unit is accepted on. Both build the PR's exact tree from a branch, so the recipe is the committed
Thor is the interesting half of the correctness question and not a consolation prize: seven of its eight CUDA feature cells resolve DISABLED for The GB10 leg is the binding one. Its acceptance test is not a ratio: the GPU busy fraction must RISE from the measured 6.31% baseline (0% in 2,019 of 2,155 samples), and the script reports the fraction WITH its sample count, because a fraction with no denominator is not a measurement. If it does not rise, that is a real result about the hypothesis and it will be reported as one. Until both legs report, |
|
Handover: what has and has not been measured, at MEASURED, x86_64 CPU-only development box (a result from it is not an A2-Q1 result, and none of it touches the device arm):
NOT MEASURED, and each is a gap rather than an omission:
Leases outstanding: If the busy fraction does not rise when it is finally sampled, that refutes the hypothesis rather than the change, and the next traceable candidate is already named in the brief and in |
…ms it divided, or refuses `examples/nemotron_h_gen` prints neither a rate nor a duration. It prints the engine load time and a `TOKEN MATCH: m/n` line, so the per-output-token number this row is measured against has to be derived from the wall clock, and a derived number that hides its terms is exactly how a rate over an unknown denominator comes to be quoted as if it had been measured. `scripts/nemotron-h-a2q1-per-token.py` prints the wall time, the load it subtracts and the token count it divides by on separate lines, and it refuses in the two cases where a plausible zero would otherwise be printed: when either term is absent from the log, and when the wall clock is not greater than the load it contains. The second case is not hypothetical -- the terms coming from different runs yields a NEGATIVE rate, which still formats like a measurement. The gate script calls it once per arm, so the device-on and device-off runs report the same shape. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…e it had to accept, so it gates the STATE Thor (sm_110) ran the focused suite and one case failed. The device arm itself passed everything else -- 49 assertions where a GPU-less box runs 4, the fresh block matched the host reference at T=1/8/12 (0.164, 0.282, 0.309 against a band of 0.5), both refusals threw, and the fp8 tower uploaded 61760 B on the first call and 0 B on the second. The failure is the instrument. The carry case banded the SECOND LEG'S OUTPUT against the separation of a dropped carry. On Thor the second leg agreed to 0.705 while a dropped carry separated by only 0.205, so the derived band (0.102) sat BELOW the deviation a FRESH leg already shows on this fixture. The two arms are W8A8 against W8A16, so a fresh leg already disagrees by the e4m3 activation quantization and a second leg compounds that with the same disagreement propagated through the carried state. A defect whose separation is smaller than the noise the comparison must accept is not resolvable from that comparison, and widening the band until it passes is what the spec's §8.1 says to stop for. So the assertion moves to what the carry actually IS. A dropped carry hands the next leg zeros, so the separation between the advanced state and a zeroed one is 1.0 by construction -- about six times the noise floor, which this fixture can genuinely resolve. The conv window and the SSM state are now banded separately, each against its own zeroed twin, each with its element count asserted against the geometry. The noise floor is MEASURED in the run, at the same width, from a fresh device leg against a fresh host leg, and printed beside the separation. The second leg's output is still measured, and it carries an assertion only when the separation exceeds twice that floor; the condition is printed either way, so "no assertion here" is a stated measurement rather than a silent hole. What this cannot see -- a carry advanced but wrong by less than the band -- is written down, and the real-checkpoint per-block gate is named as the instrument for it. Two things the failure exposed beside the band. The fixture left `mamba_ssm_cache_dtype` unset and so resolved bf16, which is NOT the configuration the paged forward selects the device arm for (`ssm_dtype == f32`); it now says float32 as the released checkpoint does, and the case asserts it, because a state silently halved is invisible to every comparison. And the whole file was a skip on a GPU-less box, so a new CPU-runnable case pins the op contract the split depends on -- three outputs of widths (I, conv_dim, num_heads) taking the head, middle and tail of an in_proj row. That case states plainly that it does not pin the production call site, which only the device numeric case covers. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
left open The spec's §3 measured Thor (sm_110) as having HALF an fp8 arm: the GEMM present through the unconditionally registered `kMatmulFp8CublasLt`, the activation quant trapped in a CUTLASS-gated translation unit that no sm_110 build compiles. #991 moved the registration out. Whether that actually reaches this model's arm was never run, and the spec said the conclusion had inverted twice during scoping, so it is worth writing down that it now has an answer. It does. On a build whose feature table reads `ENABLED for [110]: 1 ; DISABLED cells: 7` -- only `marlin-nvfp4`, with `cutlass-fp8` and both `scaledmm-c3x` cells off -- the FP8 W8A8 mamba arm executes: 49 assertions where a GPU-less box reports 4, the fresh block matching the host reference at three widths, and the fp8 tower uploading exactly once. The section also records the carry gate's failure and its repair, because the failure is the more instructive half: the case banded a defect whose separation (0.205) was SMALLER than the noise the comparison had to accept (0.164 on a fresh leg), which is §5.2's lesson arriving from the direction the spec did not anticipate. And it states which host can answer the occupancy question at all -- the 6.31% baseline is a GB10 number, so a Thor busy fraction would measure different silicon and answer something else. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…side a Thor number The A3 gate ran on Thor and printed `GPU busy in 358 of 2336 samples = 15.33% busy (baseline 6.31%)`. The denominator is right and the sample count is there, but the parenthesis is wrong: 6.31% was measured on GB10, and printing it beside an sm_110 number invites a comparison across two different pieces of silicon, two different clocks and two different memory systems. It answers a different question than the one it appears to answer. That is the failure this repository already knows by name -- a number quoted often enough comes to be treated as measured -- and a gate script is exactly where it starts, because the script's output is what gets pasted into a report. The baseline is now quoted only when `ARCH` is `121a`. On any other arch the line says so and names the comparison that IS valid there: the same-binary `VT_NEMOTRON_H_DEVICE_MAMBA` ON/OFF A/B on that same box. All three arms are self-tested, including the empty-sample arm, which reports UNMEASURED rather than 0% -- a busy fraction over zero samples formats identically to a genuinely idle GPU. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Thor (sm_110) leg: the A3 gate PASSES with the device arm onUnder an Correctness first, and it holds:
The build this ran on is the interesting part. Its feature table read The focused suite reported 49 assertions where a GPU-less box reports 4, the fp8 tower uploaded 61760 B on the first call and 0 B on the second, and the neighbouring suites were all green: forward 16/16 (5716), paged 12/12 (3256), loader 2/2, moe_device 2/2 (29), Two instrument defects this run exposed, both fixedThe busy fraction was printing The carry gate banded a defect smaller than the noise it had to accept (detail in Still owed
No per-token time came out of this run: the Thor build predates the timing helper and its script line needed |
…line it cannot be read against The A3 gate ran on Thor (sm_110) with the FP8 W8A8 mamba arm on the device and returned 96/96, mode=decode, STRICT PASS, exit 0, with zero reference-tier lines. That last number is the one worth recording beside the verdict: the portable reference tier is numerically CORRECT, so a pass obtained on it is invisible in every other figure, and its absence is the only thing that separates the two. The busy fraction was 358 of 2336 samples, 15.33%. Both surfaces say explicitly that this is NOT comparable to the 6.31% baseline the unit is accepted against, because that baseline was taken on GB10 and this is different silicon. Recording the number without that sentence would have been the more useful-looking and less true entry, and the GB10 leg is still queued. BENCHMARKS also records what the run did NOT produce: no per-token time, because the Thor build predates the timing helper and its script line needed `bc`, which the container lacks. STATUS moves from one passing gate to two and names which arm the second one exercised. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ered the wrong question (#1290) The Thor A/B ran both arms of the same binary against the same checkpoint and golden, differing only by VT_NEMOTRON_H_DEVICE_MAMBA. Two things came out of it and only one of them is a number this row may keep. THE ARM IS TOKEN-EXACT WHERE THE HOST ARM IS NOT. With the device arm on, the A3 gate read 96/96 mode=decode STRICT PASS, exit 0. With it off -- which is what main does on a GPU today -- it read 93/96 DIVERGENCE, exit 1. Zero reference-tier lines on both, so neither run took the portable tier. The mechanism is named rather than guessed: the golden comes from an oracle that computes these projections W8A8, and `DenseBf16` states outright at nemotron_h.cpp:419-422 that the host arm carries `input_scale` and does not apply it. That is #1290, filed while landing this row and fixed by it. It is n=1 per arm, and the row says so. THE BUSY FRACTION IS VOID, AND IT IS THIS UNIT'S ACCEPTANCE TEST. It read 15.33% on and 14.73% off, and neither figure means what it appears to: the sampler started with the process, so the multi-minute GPU-IDLE 20.1 GiB engine load sat inside the same window as the decode and diluted both arms toward each other. That is the same defect as summing prefill and decode into one profile, and a fraction over the wrong window is worse than no fraction, because it still formats like a measurement. No occupancy claim is made from that run in either direction. `run_gate` now starts the driver first, waits for `engine loaded in Ns`, and samples only after it; when that line never appears it reports the fraction as NOT REPORTED rather than sampling a window whose meaning is unknown. The per-token helper takes the decode window directly and subtracts nothing, since the load is already outside its brackets, and it refuses on a non-positive window -- the brackets failing to span a decode would otherwise print a negative or infinite rate that still looks like a measurement. What survives from the run without an instrument caveat is the wall-time ratio from the sample counts, one sampler at one nominal interval: 2.44x shorter with the arm on. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` moved while this row's Thor A/B was running, so the branch is merged rather than rebased and the gate is re-run over the merged tree. AGENTS.md R4: merge an immutable SHA and re-run the full gate after merging rather than reading the diff. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
The A/B is in, and it says two things — one of which is that this unit's headline number is VOIDSame lease on
The host arm is the one that diverges
★ The GPU busy fraction is VOID, and it is this unit's acceptance testIt read 15.33% on and 14.73% off. Neither figure means what it appears to. The sampler started with the process, so the multi-minute, GPU-idle 20.1 GiB engine load sat inside the same window as the decode and diluted both arms toward each other and toward zero. That is the same defect as summing prefill and decode into one profile, and a fraction over the wrong window is worse than no fraction because it still formats like a measurement. So no occupancy claim is made from this run, in either direction. The hypothesis that this arm raises GPU occupancy is neither supported nor refuted by it. What survives without an instrument caveatFrom the sample counts, one sampler at one nominal interval: the run was 2.44x shorter with the arm on (2336 vs 5702 samples). The GB10 leg is still queued and remains the only one that can answer the 6.31% question — now with a sampler that measures the right window. |
…a counter-measurement The issue as filed said the host mamba arm is what main does "on a GPU today" and therefore that main is token-wrong on a GPU. That is established for sm_110 ONLY. On GB10 (sm_121a) the SAME host arm reads 96/96 STRICT PASS -- the A3 run that closed #1157, /usr/local/nas_share/rc/nh1157/gate_fixed.out, same golden and same oracle revision. Left as written, the issue would get dismissed the first time somebody checked on GB10 and got a pass, and a silicon-specific defect would go with it. So the row, the spec and the benchmark entry now all say what was measured WHERE: host arm 93/96 on sm_110 with fa2 and cutlass-fp8 both DISABLED, host arm 96/96 on sm_121a with both ENABLED, n=1 per arm. What is established is narrower and still worth having: on ONE box, holding the entire rest of the tower constant, flipping only the mamba arm moves 93/96 to 96/96. The `DenseBf16` reading stays the leading mechanism -- the golden's oracle is W8A8 and the host arm is W8A16 because `input_scale` is carried and not applied (nemotron_h.cpp:419-422) -- but it now carries the obligation to explain why GB10 is clean, and the honest reading is that the perturbation is marginal rather than grossly wrong. One proposed candidate is excluded rather than chased. "The two arms resolve different fp8 GEMMs" cannot be the differentiator, because that configuration runs NO fp8 GEMM on either box: `DenseFor` dequantizes and hands the result to vt::MatmulBT on the CPU queue (nemotron_h_device.cpp:2027). The device-side difference that IS checkable is attention -- CudaArchFeatures.cmake:349 provides fa2 for 12.1a and not for 11.0, so GB10 runs FlashAttention-2 over the 6 GQA layers where Thor runs the portable fallback. Next step named rather than the question closed: the oracle's top-2 margin at the three moved tokens, because this family has already produced a "divergence" that turned out to be a bit-exact near-tie; then a NemotronHTrace layer bisect; then a repeat to lift n=1. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Correction to #1290: the divergence is an sm_110 result, and GB10 is a counter-measurementThe issue as filed said the host mamba arm is what
Left as written, the issue would have been dismissed the first time somebody checked on GB10 and got a pass — taking a real silicon-specific defect with it. #1290, its What is established is narrower and still worth having: on one box, holding the entire rest of the tower constant, flipping only the mamba arm moves One candidate excluded rather than chased"The two arms resolve different fp8 GEMMs" was the natural next hypothesis. It cannot be the differentiator, because the The device-side difference that is checkable is attention: Next step, named rather than the question closed
Preflight over the merged tree: **All gates green, |
…of its four behaviours ARE refusals The helper exists because `examples/nemotron_h_gen` reports neither a rate nor a duration, so the per-output-token number this row is measured on has to be derived. A derived number that prints 0, or a negative, or a rate over an unknown denominator is indistinguishable from a measurement once it reaches a report, so most of the helper is guards -- and nothing in the tree pinned them. Six cases now do: the rate is printed with the window, the excluded load and the token count beside it; the load is excluded rather than subtracted a second time; a non-positive window refuses; a missing TOKEN MATCH line refuses; zero compared tokens refuses; and the vLLM denominator is the pinned 0.014369 s rather than a constant that could drift and silently restate the gap. Armed, not merely present. Deleting the non-positive-window guard reds the suite; making the missing-token path print `per output token 0.000000 s` reds it. The tree was restored byte-for-byte between mutations and after, sha256 be32e5f5c786ccb283c88b88e256f9c0e5d79af87eeef5615a3c10a615d3ce8c both times, and the suite is green at that sha. The whole `run_gate` window change was also driven end to end against a fake driver and a fake nvidia-smi before it is allowed near the scarce GB10 lease: a 3 s load plus a 4 s decode yields 39 samples over a 3.994 s window rather than ~70 over ~7 s, so the load really is outside the brackets; a driver that dies during load reports `NO busy fraction is sampled` and still surfaces its exit code 3; and a 93/96 divergence surfaces exit 1 with `matched 93` beside the rate. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
The sampler fix is validated end-to-end before it touches the GB10 leaseA broken instrument fails toward a verdict about the code, so
The second row is the one that mattered: an instrument that swallowed the exit code while reporting a plausible fraction is exactly how an infrastructure failure comes to read as a statement about the model. The refusals are now pinned, and armed
Armed rather than present: deleting the non-positive-window guard reds the suite, and making the missing-token path print In flight
The 6.31% number is GB10's, but the hypothesis — that this arm raises the decode busy fraction — is testable by ON/OFF on any box, and Thor can answer it now. |
…se doctest stringifies char* as bool
The Thor re-run confirmed the repaired carry gate -- 5 cases, 63 assertions, 0
failed -- and printed the numbers that justify it: a noise floor of 0.2465 at
T=1, a carried conv window agreeing to 0.1746 against a band of 0.5, and a
carried SSM state agreeing to 0.0614 against the same band, at 2.9x and 8.1x
margins.
It also proved the diagnosis that drove the repair. A dropped carry separates the
second leg's output by 0.2045 while the noise the comparison must accept is
0.2465, so the defect really is smaller than the noise, `separation > 2 *
noise_floor` is false, and the case correctly asserts nothing there.
Correctly, but not legibly. The line that reports WHY no assertion was made
printed `1`. doctest stringifies a `const char*` as a bool, and the message
streamed a `char*` ternary, so it printed the truthiness of a non-null pointer
whichever branch was live -- an unconditional `1` that says nothing about the
condition it names.
That line exists to make "no assertion was made here" a STATED result rather than
a silent hole, so a version of it that cannot say what it means is the same class
of defect as the band it reports on: an instrument that formats like information.
It now builds a std::string. Reproduced against doctest 2.5.2 in both forms
first: `MESSAGE("x " << (false ? "yes" : "prose"))` prints `x 1`, and the
std::string form prints `x prose`.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
Thor re-run with the fixed sampler: the repaired carry gate holds, and its numbers justify the repair
The data now proves the diagnosis that drove the repair. A dropped carry separates the second leg's output by 0.2045, while the noise the comparison must accept is 0.2465 — the defect really is smaller than the noise, so One diagnostic was defective, and it is the kind that hidesThe line reporting why no assertion was made printed That line exists to make "no assertion was made here" a stated result rather than a silent hole, so a version that cannot say what it means is the same class of defect as the band it reports on: an instrument formatted like information. Reproduced against doctest 2.5.2 both ways before fixing — — and it now builds a Head is |
…ide a Thor decode The decode-window sampler works. With the arm on, Thor read 42.55% busy over 564 DECODE samples where the old conflated window read 15.33%, and the reason is now measured rather than argued: the engine load is 500.9 s and the decode is 75.4 s, so the old window was 87% load. The same run also printed `per output token 0.785606 s (vLLM 0.014369 s; ratio 54.7x)`. That ratio is wrong in the way the busy fraction was wrong an hour ago: 0.014369 s is a GB10 figure and this decode happened on Thor, so the quotient compares two different pieces of silicon. Fixing the busy-fraction surface and leaving its twin is how a bad comparison survives a correction, and 54.7x is exactly the kind of number that gets quoted once it exists. The rate still prints, because it is a real measurement of that box. Only the comparison is withheld, and it is withheld BY NAME rather than silently dropped, so the reader knows a ratio was declined rather than forgotten. The gate script now passes ARCH through. Two cases pin both arms, and the guard is armed: quoting the ratio unconditionally reds the suite. Tree restored byte-for-byte after the mutation, sha256 3d94497009c384bcb36709a48b3e1c5da203fea23db020ecb097b79af035cae1, green at that sha. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…n Thor, and the divergence reproduced The acceptance test of this unit was never a ratio: the GPU busy fraction had to rise, measured with its denominator. On the corrected instrument it does. Second Thor lease, fresh build and clone, same binary, same checkpoint, same golden, differing only by VT_NEMOTRON_H_DEVICE_MAMBA: arm ON 96/96 STRICT PASS rc=0 240/564 = 42.55% busy 0.785606 s/token arm OFF 93/96 DIVERGENCE rc=1 700/3808 = 18.38% busy 5.633442 s/token +24.17 points, a 2.31x rise in occupancy, and 7.17x less time per decode token. Peak host 44070 MiB. Zero reference-tier lines on either arm, so neither ran on the portable tier. READ ON THE BOX IT WAS TAKEN ON. These are sm_110 figures, and BOTH references this row is normally quoted against -- the 6.31% baseline and the 0.014369 s per token -- are GB10's. Neither supports a ratio against these numbers, the instrument withholds both by name off 121a, and the stale `ratio 54.7x` and `ratio 392.1x` strings in that run's log predate the per-token arch gate and must not be quoted. The per-token VALUES are sound; the ratios beside them are not. The divergence also reproduced, which lifts the n=1 caveat for Thor: two independent leases with separate builds agree exactly, 96/96 on and 93/96 off both times. GB10 stays n=1 in the other direction, and no GB10 run of the DEVICE arm exists at all yet -- so the hypothesis is SUPPORTED on sm_110 and UNMEASURED on sm_121a, and the owed list now says so rather than implying one covers the other. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` moved again while the second Thor lease was running. Merged rather than rebased, and the gate is re-run over the merged tree: AGENTS.md R4 says to merge an immutable SHA and re-run the full gate rather than reading the diff. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
The acceptance test is MET on sm_110: the decode busy fraction ROSE 18.38% to 42.55%Second Thor lease, fresh build and clone, sampler measuring the decode window alone. Same binary, checkpoint and golden; only
+24.17 points, a 2.31x rise in occupancy, and 7.17x less time per decode token. Peak host 44070 MiB. This is the measurement the unit was sent to produce, and it is the one the first run could not make: that run reported 15.33% vs 14.73% because its window was 87% engine load (500.9 s load against a 75.4 s decode). Fixing the window did not flatter the result — it revealed one. Read on the box it was taken onThese are sm_110 figures. Both references this row is normally quoted against — the 6.31% baseline and the 0.014369 s per token — are GB10's, so neither supports a ratio against these numbers. The instrument now withholds both by name off So: the occupancy hypothesis is SUPPORTED on sm_110 and UNMEASURED on sm_121a. The owed list says exactly that rather than letting one stand in for the other. The divergence reproduced, lifting n=1 on ThorTwo independent leases, separate builds and clones, agree exactly: Still owedThe GB10 run (the only one readable against either baseline, and the only place a device-arm A3 does not yet exist); the §5.1 per-block numeric gate against
|
`main` advanced to f06b9e9 carrying #1297, which routes `test_dspark_block_size_guard.cpp` through `tests/support/process_id.h`. This branch's `build-newest-gcc` failure was that inherited red, so it is merged forward to pick the fix up rather than re-run against a stale base. `docs/BENCHMARKS.md` was a GENUINE content conflict, not a driver artifact: this branch inserts the A2-Q1 row and main inserted the `ENG-RECORD-ANCHOR-RATCHET` row at the same point. They are different keys, so BOTH are kept -- main's first, then this row's. Neither was dropped, and no ratchet was raised to fit them. The union auto-merge of `.agents/issue-index.md` was DISCARDED: main's file taken wholesale, only rows whose KEY main lacks re-appended, main's file asserted a strict PREFIX. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…gates This branch is based on PR #1289 (`row/A2-Q1-fp8-mamba`), which is where `NemotronHMamba2MixerDevice` lives, so it did not start from `main`. The preflight's trailer and style gates SKIP when `origin/main` is not an ancestor of HEAD, and a skipped gate has reported nothing about this tree. `docs/BENCHMARKS.md` was a GENUINE content conflict, not a driver artifact: main updated the `ENG-RECORD-ANCHOR-RATCHET` row while this line carries the A2-Q1 row inserted at the same point. BOTH are kept, main's version of its own row first, then this line's. Neither was dropped and no ratchet was raised. `.agents/issue-index.md` auto-merged and was verified rather than trusted: main's file is a strict PREFIX of the result, with only rows whose KEY main lacks appended after it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ojections (#1388) The GB10 A/B varied the mamba KERNEL and concluded the 95/96 divergence was the host's. That conclusion is under-determined, and my own artefacts show why: the four kernel counters are non-zero in BOTH legs, and they are only reachable from the `mamba_on_device` branch, so both legs ran A2-Q1's FP8 W8A8 projections. The script hardcoded VT_NEMOTRON_H_DEVICE_MAMBA=1 in both. `main` carries no device mamba arm at all -- NemotronHMamba2MixerDevice, MambaIsFp8 and even the VT_NEMOTRON_H_DEVICE_MAMBA knob are absent there -- so a 95/96 on both legs is equally consistent with the divergence being the architecture's or being A2-Q1's projections, which neither leg turns off. Leg 3 routes the whole mamba block back to the host reference on the SAME binary and box, which is the only difference. Its own counters are the check that it took that path: four kernel counters at 0 with gathers and scatters non-zero, because the host branch still gathers and scatters but never enters the instrumented device mixer. A leg that cannot prove which path it took cannot discriminate anything. 96/96 on leg 3 means the divergence is A2-Q1's FP8 arm and not the architecture, which would re-scope #1388 and matters for #1289. 95/96 means the host owns it and the current framing stands. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
#1311) The GB10 A/B varied the mamba kernel and I recorded its cause as the host's. That is under-determined, and my own counters are what show it: the four kernel counters are non-zero in BOTH legs and are reachable only from the mamba_on_device branch, so both legs carried A2-Q1's FP8 W8A8 projections. The script pinned VT_NEMOTRON_H_DEVICE_MAMBA=1 in both. main carries no device mamba arm at all -- NemotronHMamba2MixerDevice, MambaIsFp8 and the VT_NEMOTRON_H_DEVICE_MAMBA knob are all absent, verified by git show on origin/main. So 95/96 on both legs fits "the host diverges" and "A2-Q1's FP8 projections diverge" equally, and this A/B separates neither. #1312's 96/96 on a main-based GB10 tree corroborates the second without settling it, being a different branch and binary. So the records stop asserting the conclusion. BENCHMARKS, the benchmark record and the spec now say the cause is undecided and name the queued discriminator: leg 3, VT_NEMOTRON_H_DEVICE_MAMBA=0 on the same binary and box, with its own counters checking it really took the host path. If it reads 96/96 the finding belongs to A2-Q1 (#1289), which carries the only real speed win measured this session, and that is a larger result than #1388 as first framed. Recording the arch reading as settled would have been the same error as reading the Thor pass across to GB10, which this row refuses everywhere else. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
#1311) The GB10 A/B varied the mamba kernel and I recorded its cause as the host's. That is under-determined, and my own counters are what show it: the four kernel counters are non-zero in BOTH legs and are reachable only from the mamba_on_device branch, so both legs carried A2-Q1's FP8 W8A8 projections. The script pinned VT_NEMOTRON_H_DEVICE_MAMBA=1 in both. main carries no device mamba arm at all -- NemotronHMamba2MixerDevice, MambaIsFp8 and the VT_NEMOTRON_H_DEVICE_MAMBA knob are all absent, verified by git show on origin/main. So 95/96 on both legs fits "the host diverges" and "A2-Q1's FP8 projections diverge" equally, and this A/B separates neither. #1312's 96/96 on a main-based GB10 tree corroborates the second without settling it, being a different branch and binary. So the records stop asserting the conclusion. BENCHMARKS, the benchmark record and the spec now say the cause is undecided and name the queued discriminator: leg 3, VT_NEMOTRON_H_DEVICE_MAMBA=0 on the same binary and box, with its own counters checking it really took the host path. If it reads 96/96 the finding belongs to A2-Q1 (#1289), which carries the only real speed win measured this session, and that is a larger result than #1388 as first framed. Recording the arch reading as settled would have been the same error as reading the Thor pass across to GB10, which this row refuses everywhere else. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
|
Converted to DRAFT to block merge while a correctness question is open. Not a review finding — a sequencing hold. WhyThe GB10
And a third GB10 run today, #1348 on a main-based tree, read
That is consistent with an architecture-specific divergence, and equally consistent with this PR's FP8 W8A8 projections losing a token on sm_121a. The existing A/B cannot separate them because neither leg turns this arm off. Different branch and binary, so #1348 favours the second reading without settling it. What settles itA third leg is queued on
Worth asking in parallel, and cheaper: the oracle's top-2 margin at the diverging token. This family has already produced a "divergence" that was a bit-exact near-tie. If the margin is tiny this may be a tie-break rather than a defect, which would also explain why Thor — with different attention arithmetic, What is NOT in questionThis PR's Thor evidence stands: A3 Mark ready again once leg 3 reports. |
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a branch touching `.agents/issue-index.md` re-conflicts whenever main appends a row. Resolved locally where the driver runs: the union auto-merge was DISCARDED, main's file taken wholesale, only rows whose KEY main lacks re-appended, and main's file asserted a strict PREFIX with zero duplicate keys. Any keyed-table collision kept BOTH sides, main's first, so no row is dropped. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
|
The discriminator reported. The hold is CONFIRMED — and the same run measured this arm as the largest speed win in the campaign. Both facts are about this PR, and neither cancels the other. Confirmed: this arm loses a token on sm_121aThree legs, one binary, one box,
Leg 3 proved it took the host path rather than merely claiming to — So the earlier Also confirmed: this arm is 6.64x faster per output token10.319 s/token -> 1.554 s/token. GPU busy 7.86% -> 10.18% on the same binary. That closes the gap against vLLM from 718.1x to 108.2x, which is by a wide margin the biggest movement measured on this model, and it is consistent with the Thor decode-window A/B already in this PR (occupancy 18.38% -> 42.55%, 7.17x less time per output token). Two hosts, two methods, same direction and similar magnitude. The arm is right. One token is wrong. That is a repair, not a rejection, and this PR should land once the token is explained. What diverges, and the cheapest next questionPositions 1-31 identical; only position 32 differs, on the longest prompt: Ask the oracle's top-2 margin at that token before writing any fix. This family has already produced a "divergence" that turned out to be a bit-exact near-tie, and a tie-break is not a defect. If the margin is real, the standing hypothesis — FP8 rounding accumulating in the CARRIED conv/SSM state, crossing a decision boundary only at the last step of the longest sequence — predicts the divergence point moves with generation length and vanishes on short generations. Testable, and currently unconfirmed. Tracking in #1388, now re-scoped from "the architecture" to this arm. Staying DRAFT until the token is explained or fixed. Thor's |
…ices that arm at 6.64x (#1311, #1388, #810) The GB10 divergence had two candidate causes and the two-leg A/B could separate neither, because both legs sat on A2-Q1's device mamba arm. A third leg removes that arm on the same binary and box: on device mamba 1 95/96 DIVERGENCE 1.584694 s/tok 110.3x off device mamba 1 95/96 DIVERGENCE 1.554233 s/tok 108.2x hostmamba device mamba 0 96/96 STRICT PASS 10.318897 s/tok 718.1x So the cause is A2-Q1's arm, not the architecture. The earlier arch-specific reading was retracted before this ran; this is the experiment that tested it. Leg 3's own counters are what make it a discrimination rather than a correlation between a flag and an outcome: state_update_rows, chunk_scan_calls, conv_update_rows and conv_fwd_calls all zero while gathers and scatters stay at 46, which is the host path's signature and nothing else's. The divergence is one token and both device legs lose the SAME one -- prompt 2, position 32 of 32, 11286 against the oracle's 3468, positions 1-31 byte-identical. The repaired verdict grep captured it; the first GB10 run threw it away. Whether that is a defect or a bf16 near-tie is NOT settled here and a fresh implementer owns it. The second result is the larger one and is not this row's: A2-Q1's arm is worth 6.64x per output token on GB10, closing 718.1x to 108.2x versus vLLM with decode busy 7.86% to 10.18%. It agrees in direction and rough magnitude with #1289's independent Thor A/B. One run per leg on a contended box, engine load excluded; the 6.64x survives that and 108.2x remains an OPEN GAP, not parity. And the +-2% on/off delta is retired as noise by a sign flip: the first GB10 run put on 1.991% FASTER, this one puts it 1.960% SLOWER, same comparison, same box. A quantity that reverses direction between two runs is not a measurement of it. That strengthens #1311's refutation, which was argued from the 3% bar alone. This row's acceptance is unchanged: each host returns the same verdict on both KERNEL arms, so the swap is token-neutral and the sm_121a token belongs to the arm underneath it. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [Claude Code]
…h axes and is still ungated, and #940 was never the live pointer (#1250, #1289, #1388) Three corrections and one confirmation, all from evidence rather than from being told. #940 has been closed since 2026-08-16, so every pointer this row carried to it named a dead issue. A2-Q1's live tracker is PR #1289, held DRAFT by #1388. The two #940 rows this repository's issue index already carried are somebody else's and are untouched; only this row's own appended row is corrected, so the append-only file keeps main as a strict prefix. The confirmation is the one this row asked for. A three-leg discriminator moved the 23 Mamba2 layers on-device and reported both halves of the prediction: warm 10.1502 -> 1.3947 s per output token, 7.28x, with the GPU-busy fraction rising 7.86% to 10.2%. Speed alone would not have tested the hypothesis; the busy fraction is what makes it more than a story. Note the warm basis: 7.28x is measured on the same cold-prompt exclusion this row uses everywhere, and a 6.64x figure for the same pair comes from including the cold prompt on both sides. It is not a parity number and the record says so in the same breath as the speedup. Both device legs read 95/96 DIVERGENCE, and AGENTS.md establishes the token-exact gate BEFORE a performance result is accepted, so 718.2x stays this row's gated figure and the device arm's ~97x-vs-oracle is carried as an ungated projection. A ceiling is still nowhere in sight either: at ~10.2% busy the decode remains roughly 90% GPU-idle, so A2-Q1 banks 7.28x and does not close the gap, and A2-Q2b's `lm_head` owes the same test. Two things went to #1388, which had neither. Its "arch- or host-specific, not arm-specific" conclusion was drawn from two device arms plus a passing Thor, with no host-arm leg on GB10; there are now three such legs and all pass 96/96 on the same binary and box, so the divergence does track the arm. And the diverging row is prompt 2, which is also the row where the pinned oracle failed to reproduce its OWN committed golden once its resolved block size moved, which is evidence for that issue's benign-near-tie branch and against a wrong recurrent carry. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…e clock-pinning spec Another session advanced this branch while the #940/#1289 pointer correction was being written. Merged rather than forced; the corrections ride on top. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] # Conflicts: # docs/BENCHMARKS.md
The flagship gate arm this spec specified could not fail. G1(c) named "one decode + one short extend" as the population that exists only because of `treat_short_extends_as_decodes=False`. It is not. A short extend has prior context by definition (`runner.cpp:144`), so it has `seq_len > 1`, and together with `query_len == 1` those are exactly the conditions of the promotion at `mamba_attn.py:450-453`. The promotion therefore always fires on a short extend and the row classifies as a decode under both modes. Traced through `utils.py:594-635`, the only population in A2-B's scope where the two modes give different four-tuples is a still-prefilling row with `query_len == 1` and no prior state, which our runner classifies as a pure first-chunk prefill at `runner.cpp:149-151`. G1(c) now has four named arms, only (c3) discriminates, and the short-extend arm is kept as (c4) because proving the promotion is ported is a distinct and worthwhile assertion. B-M1 and the blind-spot table are corrected to match, so an implementer no longer hits stop condition 3 on the unit's own flagship mutation. G0 had the same shape one level up. It is "G1 with the count clause removed", G1 has several arms, and this spec's own baseline section establishes that the per-request machinery already works on the untouched forward. So G0 as written produced no red anywhere and told the implementer a green rebutted the baseline finding. G0 now names the single arm that must red and records the others as expected green. The port map asked for one metadata field where upstream reads two. The promotion predicate reads `seq_lens_cpu_upper_bound`, not `seq_lens_cpu`, and our `backend.h:30-31` defers both. Since `seq_len > 1` is now the discriminating quantity, the substitution by our `seq_lens_cpu` is recorded as a tracked exception, sound only while speculative rows stay refused. The named `is_prefilling` producer was also unreachable at one request: the reorder classifier early-returns at `num_reqs <= 1` before computing `done_prefilling`, so the field would have been empty on every single-request step and R2 would have refused every existing NemotronH gate. The spec now says where the signal must actually come from and anchors upstream's own producer. Anchor corrections throughout, in a section that claimed every anchor was asserted unique. The promotion is `mamba_attn.py:445-461`, not `:455-462`, whose range excludes both lines quoted as its evidence. The G-SAFE clause is `nemotron_h_registry.cpp:159-167`, not `:161-170`, which is the #775 comment. A2-P's stale scope line is `:64`, not `:81`, which is a different and correct row. The `WROTE` diagnostic is `:1692-1699`, not `:1707-1714`. Re-derived myself: there are 16 call sites of `split_decodes_and_prefills`, not 13; eight pass `require_uniform` and only four of those pass a literal `True`. The conclusion the unit rests on is unchanged: `mamba_attn.py` passes it nowhere. `scripts/check-symbol-anchors.py` passes on all of these and cannot see any of them, because they are bare line ranges carrying no symbol. Also: D4's ordering assertion re-derives the classification from the classifier it checks, so it holds by construction on an ordered batch and is not what asserts D-B1; the mutation report's binary sha256 is necessary and never sufficient because this build is not byte-reproducible; D6's throughput prediction names its two rates and a spread rule so "did not improve" is decidable; #940 is marked closed; and an unescaped pipe that had been silently splitting a table cell is escaped. D6 had the correctness-first polarity inverted. It said the gated 718.2x baseline was dead and treated 108.2x as the live denominator, but BOTH of A2-Q1's device legs fail their token gate at 95/96 and only the host arm passes 96/96, so the device figure is an ungated projection that cannot supersede a gated baseline. AGENTS.md requires the token-exact gate before a performance result is accepted, which is why #1289 is DRAFT. 718.2x is restored as the gated figure, the ratio is corrected to 7.28x on a stated warm basis, and the confusable "arm on / arm off" labelling is replaced by naming the arm, because the leg named a3_off is itself a device leg. At ~10.2% GPU busy the decode stays ~90% GPU-idle, so A2-Q1 banks a multiple and does not close the gap. The issue-index row is amended on the branch rather than appended to, which is legitimate and is the only window: the row does not exist at the merge base, so `check-issue-index-append-only.py` sees a pure append and the correction can never be made once the row lands. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
2 participants
feat(#810 A2-Q1): NemotronH's 23 Mamba2 blocks reach the device on the FP8 W8A8 seam (#517)
The 23 Mamba2 layers were the last host bounce of a decode step, and they were
the expensive one. The host arm reaches its two projections through
Linear(..., const NemotronHOwned&), whoseDenseForcallsNemotronHOwned::DenseBf16()-- a FULL dequant of the fp8 tower into a freshbf16 buffer ON EVERY CALL. That is 23 x (10304x2688 + 2688x4096) = 890e6
elements re-expanded per token, plus a download of the normed hidden and an
upload of the mixer output per layer.
The block moves whole or not at all:
mixer.in_projproduces the fusedzxbcdtthat the causal conv and the SSD scan both consume (
mamba_mixer2.py:550, split:692-696), so there is no intermediate landing in which the conv is on thedevice and
in_projis not. That is why the shared FP8 W8A8 linear seam had tobe extracted first (#940,
dense_fp8_gemm.h), and this change routes through itrather than re-typing its entry points.
What it does
NemotronHMamba2MixerDevicemirrors the host arm statement for statement --same vt:: ops, same order, same dtypes, different backend:
The one substitution is the split: the host arm copies columns with
SliceColsbecause
vt::Mamba2ChunkScanvalidates every operand contiguous, andvt::QkvSplitis exactly that copy on the device.Both forwards select it at runtime, never by preprocessor.
MambaIsFp8namesonly
NemotronHWeightFormandvt::OpRegisteredis the op table's own answer,so a dense NemotronH or a device without the fp8 pair keeps the host bounce.
The paged arm hands the mixer the recurrent rows
vt::GdnStateGatheralreadygathered and zeroed and it advances them IN PLACE, so the two downloads, the
host mixer and the two uploads per layer are gone; A2-P's gather/scatter pair
and its
has_initial = truereasoning are unchanged.The
ssm_dtype == f32term in the paged selection is not decoration. The gatherwidens the page into an f32 working buffer by op contract and the host arm then
narrows it back to
ssm_dtype, so on a checkpoint whosemamba_ssm_cache_dtypeis not f32 the two arms would round differently and the per-block numeric gate
would compare two different computations.
Three points where this differs from the spec, deliberately
Fp8Weightpair is built ON FIRST DEVICE USE in aResidentSlottheweights own, not by the loader as spec §4.1/§4.2 proposed. That is A2-Q2a's
newer idiom and it is better here: it does not double the 890 MB tower in
host memory at load, it uploads nothing on a host-only run, and it leaves
rep.host_bytes(the literal18888922112pinned attest_nemotron_h_loader.cpp:310) untouched, so §4.2's re-derivationobligation does not arise. Keying on the slot rather than an address is
CUDA BF16 MoE resident cache reuses stale pointers after engine teardown #237: across two engine builds in one process an address-keyed cache returns
the previous engine's device pointers, which is plausible wrong values.
dense_fp8::ResidentFp8still does not callload_stats::AddDeviceUpload--that is FP8 W8A8 weights are uploaded without load accounting or the post-upload residency step, unlike every other resident weight in the same file #974, unchanged, and the shared header is not touched -- so A2-Q1
accounts what IT uploads at the site that causes it, as
ResidentWeightandResidentNvfp4do.ResidentFp8has uploaded it,so the conversion peaks at one projection rather than at the whole tower.
The comparison is not bit-comparable, and that is the point
The host reference is W8A16:
DenseBf16says outright thatinput_scaleiscarried and not applied. The device arm is W8A8, which is what vLLM does. The
difference between them is the e4m3 activation quantization, so every band in
tests/vllm/models/test_nemotron_h_mamba_device.cppis measured in the runagainst a defect the fixture separates, and every element count is asserted
against the geometry -- a maximum over zero elements is also 0.0.
The new file is the cheap arm in front of the real-checkpoint gate, for the same
reason A2-Q2a's device MoE file exists:
BuildTinyis all-kDense, so nothingin the existing device suite could reach the FP8 W8A8 arm at all.
G-SAFE
Untouched. All three clauses of the interlock at
nemotron_h_registry.cpp:162stand. This change creates no paging, no carriedstate and no batching that A2-P did not already create, and
num_reqs <= 1remains A2-B's to remove.
What is NOT proved here
The §5.1 per-block numeric gate on the real 20.1 GiB checkpoint, the A3
96/96 mode=decode STRICT PASSre-run, the §5.3 mutation pass and the GPU busyfraction are the unit's acceptance test and they need a GB10 lease.
scripts/nemotron-h-a2q1-dgx-gate.shrecords the recipe, including the threeenvironment facts that have each voided a run before: the CUDA lane is
sbsa,nvcc --versionis not a sufficient postcondition, and/workspaceis CIFS andrefuses symlinks. The spec's
## Nowand## Owedsay so, anddocs/BENCHMARKS.mdrecords the measurement as PENDING rather than as silence.Issue: #810 (parent row #517). Spec:
.agents/specs/nemotron-h-a2q1-fp8-mamba.md.FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]